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ABSTRACT 

A classroom response system (CRS) involves the use of remote devices allowing all 
students in a class to respond to questions displayed on a projection screen. A CRS instantly 
collects, tallies, and shows students' answers to a question on a classroom projection screen—a 
similar system used in the TV show "Who wants to be a millionaire" to collect audience 
responses. In this paper, the author proposed a comparable technology—Google Forms—that 
can be used in a computer lab and then studied gender differences in the relationship between the 
use of Google Forms and performance changes. Performance was measured by both the letter 
grades distribution and raw exam scores. The initial results suggested that the use of Google 
Form did not make any difference in the performances measured by the letter grades distribution. 
However, when the raw exam scores were compared with a covariate of calculus grades, the 
results suggest that the use of Google Forms resulted in improved performance. Further, the 
performance improvement of male students was greater than that of female students. Discussions 
and areas for future research are also presented. 

Keywords: classroom response system, gender, performance, calculus, ANCOVA, Google 
Forms 
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INTRODUCTION 

To improve student learning, various technologies have been developed and used in 
colleges and universities, from overhead projectors, to LCD projectors, online testing, and 
classroom response systems (CRS) (MacGeorge et al., 2007). In a CRS, students respond to 
questions posed by an instructor using a handheld device known as a clicker. Input capability of 
a typical clicker is very limited—only a few buttons representing letters (e.g., A through E) 
(Hanley & Jackson, 2006). As a result, the questions used in a CRS are mostly limited to true- 
false or multiple choice questions. A CRS instantly collects, tallies, and shows students' answers 
to a question on a classroom projection screen—a similar system used in the TV show "Who 
wants to be a millionaire " to collect audience responses (Caldwell, 2007). 

Benefits of using a CRS in classes are well documented in academic research (See, for 
example, d’ Inverno, Davis, & White, 2003; Hatch, Jensen, & Moore, 2005; Wood, 2004). One 
important benefit of using a CRS is that it makes a class interactive with longer sustained 
attention from students and thereby motivates students to participate in class discussion (Stowell 
& Nelson, 2007) and activates students' thinking (Collins, 2007). Through interactive sessions by 
using such a system, instructors can gauge students' understanding of materials during class and 
provide prompt feedback (Caldwell, 2007; Martyn, 2007; Trees & Jackson, 2007). These 
benefits of a CRS are also related to five of Chickering and Gamson's (1987) seven principles for 
good practice in undergraduate education: active learning, prompt feedback, student-faculty 
communication, and cooperation among students. The recent popularity of a CRS, or clickers, in 
college classrooms is then at least partly due to these reported benefits. 

In this study, the author proposes a computer technology—Google Forms—that may 
supplant a CRS in a computer lab. Therefore, Google Forms, like a CRS, can be used in class to 
aid students' learning and to promote a more interactive classroom environment. Google Forms 
is a free service from Google, which emulates an on-line survey system. It supports various types 
of questions: text, paragraph, multiple choice, choose from a list, checkboxes, scale, and grid. A 
text type is used for short text answer questions and a paragraph type for longer text answer (or 
essay) questions. In both types, respondents can type any text they want, unlike a typical CRS. 
Multiple choice and choose from a list are for questions in which respondents are allowed to 
choose only one item as their answer. A checkboxes type, on the other hand, allows respondents 
to choose more than one item. Scale and grid types are for Likert-scale questions. By using one 
or more question types, instructors can make a set of questions which are then posted on 
webpages. Students access these webpages using their computer workstations in a lab classroom. 
They choose or type their answers using a keyboard and mouse. This process is similar to that of 
filling out on-line questionnaires. 

Many reported benefits associated with the use of a CRS are often based on the students' 
perception. However, such perceived benefits may not always be realized as performance 
improvement, including higher exam scores (Draper & Brown, 2004; Judson & Sawada, 2002). 
King and Josh (2008) studied students' performance on exams. They found that active 
participation in the clicker response questioning often led to better performance, and the effect 
was stronger for male students. They also found that female students used clickers more actively. 
On the contrary, Morgan (2008) found the use of clicker neither improved the grade distribution 
nor lowered the attrition rate (i.e., percentage of students who dropped the course). In fact, the 
grades of the classes using clickers were worse than those of the classes not using clickers even 
though the difference was not statistically significant. 
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These findings suggest that, while the incorporation of Google Forms or clickers creates 
an opportunity for all students to participate actively in the learning process, there is no 
guarantee every student will benefit from the technology. Many studies pointed out gender 
difference in utilizing classroom technologies (Enoch & Soker, 2006; Joanna L., 1999; Kang, 
Lundeberg, Wolter, delMas, & Herreid, 2011; Robin H., 2009; Sullivan, 2001). For example, 
Enoch and Soker (2006) found male college students are more comfortable with online course 
components. Further, gender difference may exist in certain disciplines regardless of 
technologies used. For example, female students often perform worse than their male 
counterparts in the STEM (science, technology, engineering, and math) disciplines (Coley, 2001; 
Kahl, Fleming, & Malone, 1982; Zerega, Haertel, Tsai, & Walberg, 1986). However, there are 
other studies reporting no significant gender difference in STEM disciplines (Becker & Chang, 
1986; Shepardson & Pizzini, 1994). 

The goals of this paper are two folds. The first is to introduce Google Forms and to 
explain its use in class. The second and the main focus is to investigate the effectiveness of 
Google Forms by genders to see if its use follows a similar pattern to previous clicker research. 

METHODOLOGY 

The author has been teaching business statistics in a computer lab at an AACSB 
accredited business school in the southeast US. To promote an interactive class, he introduced 
Google Forms to his class in spring 2010. Class began with a lecture of about 10 to 15 minutes. 
During the lecture, main concepts of that day's class were briefly covered along with some 
example questions. For instance, the definitions of mean, median, and mode were explained. 
Then, using textbook examples, the instructor explained how to calculate such measures by using 
Excel. After the lecture, students were asked to access the webpage containing relevant Google 
Forms questions and to work on them individually or with their immediate neighbors as a group. 
Once students submitted their answers, the classroom projection screen showed their answers, 
which the instructor discussed. After discussion of the answers, the instructor moved to another 
topic and started a new cycle. That is, he lectured new topics for 10 to 15 minutes, asked students 
to answer the Google Forms questions, and discussed the their answers. More detailed 
description of how the author used Google Forms can be found in_. 

Data were collected from 81 students who registered for business statistics courses in fall 
2009 (F09) and spring 2010 (S10). There were two regular (i.e., non-online) sections in each 
semester. Google Forms was not used in class in F09, but was implemented in S10. Of the 81 
students, seven students withdrew from the courses. The numbers of male and female students 
per semester are shown in Table l(The numbers in the parentheses are the number of students 
who withdrew from the course.). 

The business statistics at the author’s institution requires a calculus course as a 
prerequisite. That is, all students in the sample had taken a calculus course and passed with a 
grade of C or better before they registered for the business statistics course. As a control variable, 
the mean grade points" for the calculus course for all student (i.e., ignoring the gender) were 
compared between semesters (see Table 2), and there was no significant difference (t = 0.3063, 
df= 79, p = 0.6199). Further, the mean grade point of female students in F09 was compared with 


1 Self-citation. Omitted for review 
Z A = 4, B = 3,C = 2 
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that of female students in S10, and no significant difference was found (t = -0.4926, df= 28, p = 
0.3131). Similarly the mean grade point of male students in F09 was compared with that of male 
students in S10, and again no significant difference was found (t = 0.9784, df= 49, p = 0.8337). 

To determine the effectiveness of Google Forms in the statistics class, distributions of the 
letter grades earned by students in F09 and S10 were compared. The grade distributions are 
shown in Table 3. At first glance, the grade distribution of S10 appeared to be worse than that of 
F09 as fewer A's and more C's were given in S10. The chi-square test revealed, however, that 
there was no significant difference (y 1 = 4.06, df= 5, p = 0.54). Additionally, the grade 
distributions of the two semesters were compared by gender. Neither female nor male students 
showed significant differences between F09 and S10 (female: y 1 = 5.28, df= 5, p = 0.38; male: y~ 
- 4.95, df= 5, p = 0.42). 

Even though Google Forms was used throughout in S10, it was utilized most heavily on 
the materials for the second exam (henceforth, the exam), which covered probability and 
probability distributions. According to the raw scores, this exam was the most difficult exam 
among all exams in author's statistics class. ’ Among the 81 students, 77 students (95%) took the 
exam. 

The mean scores on the exam for both male and female students are shown in Table 4. 
When the overall mean score of F09 and that of S10 was compared, there was no significant 
difference (t = -0.5492, df= 75, p=0.2923) even though the mean score of S10 was slightly 
higher than that of F09. This suggests that Google Forms did not improve the students' exam 
score significantly when both genders were combined. 

To assess whether or not there was a gender effect, a general linear model (GLM) was 
run with the exam score as the dependent variable. A calculus grade point was used as a 
covariate. Independent variables were gender (0 = female, 1 = male), semester (0 = Fall 09, 1 = 
Spring 10), and their interaction. The model is shown below: 

Exam 2 score = /?o + /hCalculus + /^Gender + /^Semester + /CGcnderxSemester + c 

The analysis revealed that the calculus grade was positively related with the exam score 
(J3 1 = 5.642, p = .004). It also showed that the interaction between gender and semester (i.e., 
Google Forms) was significant (p = .031). The graph in Figure 1 shows that Google Forms is 
more effective for male students, which means the simple effect of Google Forms was greater for 
male students. Note that without controlling for the prerequisite calculus grades, the mean scores 
on the exam for female students, as shown in 

Table 4, indicated that Google Forms actually worsened female students' score in S10. 
However, when the exam scores were controlled for the calculus grades, female students' grades 
were higher in S10 with Google Forms (Figure 1). 

The significant main effect of semester (i.e., Google Forms) indicated that the mean exam 
scores of S10 was greater than that of F09 when the gender was ignored and the scores were 
controlled for the calculus grade. Therefore, the analysis suggests that Google Forms is effective 
overall, and its effectiveness is higher for male students. 


3 The final exam was not cumulative. 
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DISCUSSION 

The purpose of this study was to examine gender differences in the relationship between 
the use of Google Forms and performance improvement. The present study suggests that the 
students' performance was better when Google Forms was used. This result is encouraging in 
that Google Forms can be a zero cost replacement for a CRS if a class is taught in a computer lab. 
Its positive effect on the performance, however, was not initially obvious. 

When the grades for courses (Table 3) were compared between the two semesters, there 
was no significant difference. That is, the use of Google Forms did not result in the performance 
improvement as measured by grade distribution. Further, there was no significant gender effect. 
However, it should be noted that the letter grades for the course can be influenced by non¬ 
objective factors, such as curving, end-of-semester extra credit and class participation score. 

To minimize the effect of such non-objective factors on the study, non-curved scores of 
the exam were compared between the two semesters. The results of the GLM analysis with 
gender and calculus grades suggested that the effect of Google Forms was positive, that is, it 
increased the students' performance. More data are needed to determine, however, if this result is 
applicable to other student populations or to other courses. 

An interesting finding of this study is that Google Forms helped male students more than 
female students. Even though the reason for this is not clear, one may speculate that such a 
discrepancy is at least partly due to the nature of the statistics course. There are many studies that 
point out male students perform better than female students in the STEM disciplines (Coley, 

2001; Kahl et al., 1982; Zerega et al., 1986). There are some findings in the literature indicating 
that male students may be able to take advantage of classroom technology, such as Google 
Forms, better than female students (King & Joshi, 2008). Further, male students may have 
perceived the usefulness of a CRS, such as Google Forms, more than female students (Kay, 

2009). 

However, the lesser effect of Google Forms for female students found in the current 
study may be simply from poor participation or adoption from female students, which cannot be 
tested since the current study did not collect the information on the frequency of usage of Google 
Forms by gender. King and Josh (2008), who studied the effectiveness of clickers, also found 
similar results and determined the positive relationship between active participation and 
improved performance was stronger for male students than for female students. 

In adopting information technology, the gender difference has been investigated and 
found to be significant in many studies. For example, Venkatesh and his colleagues (2003) found 
the positive effect of perceived usefulness of the information technology on intention to adopt is 
moderated by gender and the effect is greater for male; and the positive effect of perceived ease 
of use of the information technology on intention to adopt is also moderated by gender and the 
effect is greater for female. If the female students participated less frequently in the current study, 
it could be due the perceived difficulty of use of Google Form. To test such a claim, the future 
studies should investigate whether or not female students are less likely to adopt Google Forms 
than male students through the lens of other theories, such as the technology adoption model. 

Another avenue for future study is to investigate the use of Google Forms in online 
courses and its effectiveness—especially because comparable technologies are few in online 
classrooms. Google Forms utilizes webpages to post questions and they can be accessed as long 
as there is a computer device, including smartphones and tablets, with Internet access. One 
student in the author’s class commented, "The best part of this technology was that it was 
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available from home or school or any remote place." Since instructors cannot monitor students in 
online classrooms as well as in regular classrooms, they often rely heavily on various out-of¬ 
class assignments to ensure that students follow class materials. Such many assignments can be a 
burden to students as well as to an instructor. Comparison between an on-line course with and 
without Google Forms can provide an insight on whether one can reduce the number of 
assignment, if desired, without hurting student performance. 
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APPENDIX 

Table 1. Sample demographics by semester and gender 



Female 

Male 

Total 

Fall 2009 

12 

25 

37 

(1) 

(1) 

(2) 

Spring 2010 

18 

26 

44 

(2) 

(3) 

(5) 

Total 

30 

51 

81 

(3) 

(4) 

(7) 


Table 2. Mean grade point of the pre-requisite calculus course on a 4.0 scale 



Female 

Male 

Overall 

Fall 2009 

2.75 

2.72 

2.73 

Spring 2010 

2.89 

2.54 

2.68 


Table 3. Grade distribution of the sample 


Grades 

Gender 

A 

B 

C 

D 

F 

w 

Total 


Female 

4 

3 

■P 


0 

0 

11 

Fall 2009 

Male 

4 

11 

7 

l 

1 

1 

25 


Total 

8 

14 

10 

2 

1 

2 

37 


Female 

1 

8 

3 

3 

1 

1 

17 

Spring 2010 

Male 

3 

6 

14 

0 

0 

0 

24 


Total 

4 

14 

17 

3 

1 

5 

44 


Table 4. Mean scores on the exam 



Female 

Male 

Overall 

Fall 2009 

67.2 

61.0 

62.9 

Spring 2010 

61.3 

67.1 

64.7 
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Table 5. General linear model analysis output 


Source 

Type 111 SS 

df 

Mean Square 

F 

P-value 

Corrected Model 

5787.422 

4 

1446.855 

11.082 

.000 

Intercept 

7117.999 

1 

7117.999 

54.520 

.000 

Calculus 

1132.515 

1 

1132.515 

8.674 

.004 

Gender 

61.614 

1 

61.614 

.472 

.494 

Semester 

3138.073 

1 

3138.073 

24.036 

.000 

Gender x Semester 

643.481 

1 

631.481 

4.837 

.031 

Error 

9400.169 

72 

130.558 



Total 

257369.444 

77 




Correct Total 

15187.590 

76 





R 2 = .381 


Figure 1. Estimated marginal means of exam 2 

70.0 i 
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